1. IMPORTING DATA & BASIC INFORMATION¶
In [1]:
# importing libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.express as px
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
# importing data
data = pd.read_csv('hotel_cancel.csv')
In [2]:
# data shape
print("data rows & columns ->",data.shape)
In [3]:
# data's top 5 rows
data.head()
Out[3]:
FEATURE INFORMATIONS¶
- hotel: ResortHotel (H1) or CityHotel (H2)
- is_canceled: Value indicating if the booking was canceled (1) or not (0)
- lead_time: Number of days that elapsed between the entering date of the booking into the PMS and the arrival date
- arrival_date_year: Year of arrival date
- arrival_date_month: Month of arrival date
- arrival_date_week: Week number of year of arrival date
- arrival_date_day: Day of arrival date
- stays_in_weekend_nights: Number of weekend nights (Saturday or Sunday) the guest stayed or booked to stay at the hotel
- stays_in_week_nights: Number of week nights (Monday to Friday) the guest stayed or booked to stay at the hotel
- adults: Number of adults
- children: Number of children
- babies: Number of babies
- meal: Type of meal booked. Categories are presented in standard hospitality meal packages: Undefined/SC – no meal package; BB – Bed & Breakfast; HB – Half board (breakfast and one other meal – usually dinner); FB – Full board (breakfast, lunch and dinner)
- country: Country of origin. Categories are represented in the ISO 3155–3:2013 format
- market_segment: Market segment designation. In categories, the term “TA” means “Travel Agents” and “TO” means “Tour Operators”
- distribution_channel: Booking distribution channel. The term “TA” means “Travel Agents” and “TO” means “Tour Operators”
- is_repeated_guest: Value indicating if the booking name was from a repeated guest (1) or not (0)
- previous_cancellations: Number of previous bookings that were cancelled by the customer prior to the current booking
- previous_bookings_not_canceled: Number of previous bookings not cancelled by the customer prior to the current booking
- reserved_room_type: Code of room type reserved. Code is presented instead of designation for anonymity reasons
- assigned_room_type: Code for the type of room assigned to the booking. Sometimes the assigned room type differs from the reserved room type due to hotel operation reasons (e.g. overbooking) or by customer request. Code is presented instead of designation for anonymity reasons.
- booking_changes: Number of changes/amendments made to the booking from the moment the booking was entered on the PMS until the moment of check-in or cancellation
- deposit_type: Indication on if the customer made a deposit to guarantee the booking. This variable can assume three categories: No Deposit – no deposit was made; Non Refund – a deposit was made in the value of the total stay cost; Refundable – a deposit was made with a value under the total cost of stay
- agent: ID of the travel agency that made the booking
- company: ID of the company/entity that made the booking or responsible for paying the booking. ID is presented instead of designation for anonymity reasons
- days_in_waiting_list: Number of days the booking was in the waiting list before it was confirmed to the customer
- customer_type: Type of booking, assuming one of four categories: Contract - when the booking has an allotment or other type of contract associated to it; Group – when the booking is associated to a group; Transient – when the booking is not part of a group or contract, and is not associated to other transient booking; Transient-party – when the booking is transient, but is associated to at least other transient booking
- adr: Average Daily Rate as defined by dividing the sum of all lodging transactions by the total number of staying nights
- required_car_parking_spaces: Number of car parking spaces required by the customer
- total_of_special_requests: Number of special requests made by the customer (e.g. twin bed or high floor)
- reservation_status: Reservation last status, assuming one of three categories: Canceled – booking was canceled by the customer; Check-Out – customer has checked in but already departed; No-Show – customer did not check-in and did inform the hotel of the reason why
- reservation_status_date: Date at which the last status was set. This variable can be used in conjunction with the ReservationStatus to understand when was the booking canceled or when did the customer checked-out of the hotel
In [2]:
# data's summary
data.info()
2. NULL VALUE ANALYSIS¶
In [11]:
# data overview & samples (especially on nan & nan percentage)
listItem = []
for col in data.columns :
listItem.append([col, data[col].dtype, data[col].isna().sum(), round((data[col].isna().sum()/len(data[col])) * 100,2),
data[col].nunique(), list(data[col].drop_duplicates().sample(2).values)]);
dfDesc = pd.DataFrame(columns=['Data Features', 'Data Type', 'Null Count', 'Null %', 'N-Unique', 'Unique Sample'],
data=listItem)
dfDesc
Out[11]:
In [12]:
# agent (null 14%) feature unique counts
data["agent"].value_counts()
Out[12]:
In [13]:
# company (null 94%) feature unique counts
data["company"].value_counts()
Out[13]:
In [14]:
# country (null 0.4%) feature unique counts
data["country"].value_counts()
Out[14]:
In [16]:
# rows with null
len(data[data.isnull().any(axis=1)])
Out[16]:
In [19]:
# rows without null
len(data[~data.isnull().any(axis=1)])
Out[19]:
In [24]:
# trying to find the null combinations
pd.options.display.max_rows = None
null = []
for idx in range(len(data)):
temp = ""
for i in data.columns:
if (str(data[i].iloc[idx]).lower() == 'nan'):
temp += i+" "
else:
pass
null.append(temp)
(pd.Series(null)).value_counts()
Out[24]:
From the observations:
- It turns out that 'agent' & 'company' columns has more than 10% null values, where 'company' even has at least 94% null value. These columns has so much null rows that perhaps it is safer to drop them. Moreover, the null rows & columns appears to be at random, thereby creating noise. To avoid bias (by imputing many null values with speficied inputs), we could first drop 'agent' & 'company' columns and then drop the remaining null rows.
In [89]:
# creating copied data for contigency
mydata = data.copy()
# dropping columns & then the remaining null rows
mydata = mydata.drop(['agent','company'], axis=1)
mydata = mydata.dropna()
# resetting index
mydata.reset_index(drop=True)
# data's new shape
print("before ->",data.shape)
print("after ->",mydata.shape)
In [8]:
# final null values checking
mydata.isnull().sum()
Out[8]:
3. EXPLORATORY DATA ANALYSIS¶
In [40]:
# basic description for numerical features
pd.options.display.max_rows = None
mydata.describe().T
Out[40]:
In [14]:
# basic description for numerical features based on canceled or not
pd.options.display.max_rows = None
mydata.groupby('is_canceled').describe().T
Out[14]:
In [5]:
# basic description for categorical features
pd.options.display.max_rows = None
mydata.describe(include="object").T
Out[5]:
In [6]:
# basic description for categorical features based on canceled or not
pd.options.display.max_rows = None
mydata.groupby('is_canceled').describe(include="object").T
Out[6]:
In [6]:
# basic description for targer feature ("is_canceled")
plt.title("Order Cancellation Count", fontdict={'fontsize': 15})
sns.countplot(data = mydata, y ='is_canceled')
Out[6]:
3a) Pearson Correlation Analysis¶
In [18]:
# plotting pearson correlation heatmap
plt.figure(figsize=(20,20))
sns.heatmap(mydata.corr(), annot=True)
Out[18]:
In terms of Pearson Correlation:
- There seems to be a moderate positive linear relationship between lead_time (days between booking & arrival date) & is_canceled (order cancellation) feature
- There seems to be a weak positive linear relationship between previous_cancellations (amount of previous canceled orders) & is_canceled feature
- There seems to be a weak negative linear relationship between total_of_special_requests (amounts of special requests made) & is_canceled feature. The same relationship also apparent between required_car_parking_spaces (amount of parkings space required) is_canceled feature.
- There seems to be a very weak or no linear relationship at all between the rest of the features & is_canceled feature (note that I dont include booking_changes because it's a feature of potential leakage)
3b) Spearman Correlation Analysis¶
In [19]:
# plotting spearman correlation heatmap
plt.figure(figsize=(20,20))
sns.heatmap(mydata.corr("spearman"), annot=True)
Out[19]:
In terms of Spearman Correlation:
- There seems to be a moderate positive monotonic relationship between both lead_time & previous_cancellations on is_canceled feature
- There seems to be a weak negative monotonic relationship between features such as total_of_special_requests, required_car_parking_spaces, previous_bookings_not_canceled (amounts of stayed orders) on is_canceled feature
- There seems to be a very weak or no monotonic relationship at all between the rest of the features & is_canceled feature
3c) Cramer's V Analysis¶
In [13]:
# defining cramers v function to see the association between two categorical features
def cramers_v(x,y):
import scipy.stats as ss
confusion_matrix = pd.crosstab(x,y)
chi2 = ss.chi2_contingency(confusion_matrix)[0]
n = confusion_matrix.sum().sum()
phi2 = chi2/n
r,k = confusion_matrix.shape
phi2corr = max(0, phi2-((k-1)*(r-1))/(n-1))
rcorr = r -((r-1)**2)/(n-1)
kcorr = k -((k-1)**2)/(n-1)
return np.sqrt(phi2corr/min((kcorr-1),(rcorr-1)))
In [14]:
# cramer's v value for each categorical columns towards target variable
categorical_columns = mydata.select_dtypes('object').columns
print ("Cramer's V")
for i in categorical_columns:
print("-"+i+":\n {}".format(cramers_v(mydata[i], mydata['is_canceled'])))
In terms of Cramer's V:
- There seems to be a moderate association in features such as country (country origin), market segments (market segmentation; just like distribution_channel but more detailed), customer_type (types of customer), deposit_type (types of deposit) on our target variable of is_canceled feature
- There seems to be a weak association between hotel (hotel type) & is_canceled feature
- There seems to be no association between the rest of the features & is_canceled feature (assigned_room_type, reservation_status, reservation_status_date are features leaking is_canceled information)
3d) On Market Segments¶
In [101]:
# total bookings per market segment
mysegments1 = mydata[mydata['is_canceled']==1]['market_segment'].value_counts()
mysegments2 = mydata[mydata['is_canceled']==0]['market_segment'].value_counts()
# pie plot for canceled orders
fig1 = px.pie(mysegments1, values= mysegments1.values, names=mysegments1.index,
title="Booking per market segment for canceled orders")
fig1.update_traces(rotation=-90, textinfo="percent+label")
fig1.show()
# pie plot for stayed orders
fig2 = px.pie(mysegments2, values= mysegments2.values, names=mysegments2.index,
title="Booking per market segment for stayed orders")
fig2.update_traces(rotation=-90, textinfo="percent+label")
fig2.show()
From the observations:
- In terms of market segment, both canceled & stayed orders are for the largest part generated by Online TA. This makes sense due to the increasing presence & usage of Online TA's by customers (as it's easier for them to compare & book hotels) & hotels (as it's easier for them to showcase their facilities & receive popular tractions)
- What's quite interesting is that for the canceled orders, the Groups segment is significanty larger than their counterpart (stayed orders) in terms of relative size (27.4% compared to 10.3%) & order counts (12,097 compared to 7,709). This goes as well for the Direct segment where their canceled orders are significantly smaller in terms of relative size (4.35% compared to 14.1%) & order counts (1,920 compared to 10,528).
3e) On Repeater Guests¶
In [133]:
sns.set(style = "whitegrid")
plt.figure(figsize= (18,5))
# countplot for repeated guest count
plt.subplot(1,3,1)
plt.title("Repeater Guests Count", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="is_repeated_guest")
# countplot for cancellation count wth repeated guest hue
plt.subplot(1,3,2)
plt.title("Cancellation Count by Repeater Guests", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="is_repeated_guest", hue='is_canceled')
# barplot for repeated guest & cancellation
plt.subplot(1,3,3)
plt.title("Cancellation Mean by Repeater Guests", fontdict={'fontsize': 15})
sns.barplot(data=mydata, y="is_canceled", x='is_repeated_guest')
Out[133]:
From the observations:
- In terms of repeater guest count, it is not surprising that there is much less repeater guests than non-repeater guests. For repeater guests are loyal customers that's hard to come by or cultivated. They surely need to be treated well & exclusively so that they shall remain as loyal customers.
- In terms of cancellation counts, It is not surprising as well to see that more cancellation counts are from non-repeater guests than repeater guests. That account is surely represented in the next plot that shows non-repeater guests have higher mean towards cancellation (closer to value of 1) than repeater guests with at least 0,3 mean differences. In other words, non-repeater guest has a higher average to cancel than repeater guests.
3f) On Lead Time¶
In [82]:
# plot for highest occuring lead time
mydata['lead_time'].value_counts().head(10).plot(kind="bar")
Out[82]:
In [130]:
# percentage of top 2 most frequent values
print("Percentage of last-minute bookers:",sum(mydata['lead_time'].value_counts(normalize=True).head(2).values))
# percentage of the rest of values
print("Percentage of early-bookers:",sum(mydata['lead_time'].value_counts(normalize=True).values)-sum(mydata['lead_time'].value_counts(normalize=True).head(2).values))
In [116]:
# plotting kdeplot with cancellation as hue
sns.FacetGrid(data, hue ='is_canceled', height=5, xlim=(0,700)).map(sns.kdeplot,'lead_time', shade = True).add_legend()
Out[116]:
In [102]:
# scatter plot of lead time, previous cancellations & order cancellation
fig = px.scatter(mydata, x='lead_time', y='previous_cancellations', color='is_canceled',
title="Lead-time, Previous Cancellations & Order Cancellation")
fig.update_layout(autosize=False,width=700,height=500,margin=dict(l=50,r=50,b=100,t=100,pad=4))
fig.show()
From the observations:
- In terms of lead time (days) value count, it seems that the highest count is still achieved by 0 lead time or same-day bookings, followed by 1 lead time or previous-day bookings. In other words, these categories are last-minute bookers. Hotels may differ about the threshold but usually, customers with more than 1 day lead time could be considered as early bookers.
- It appears that 91,9% of them are early bookers & only 8% are last-minute bookers. This may turn out to be interesting since the early bookers, according to the KDE plot, have a larger propensity to cancel their bookings once its reaches >=50 days lead time. Based on their previous descriptions, we could also see that the group mean & median for stayed orders are 80 & 40 days (in lead time), respectively. It is surely far smaller than the group mean & median for canceled orders of 114 & 113 days, respectively. More indication that the longer the lead time, the bigger amount of order cancellations.
- If we connect the plots with previous cancellations feature, we could also see that the higher previous cancellations a customer has, especially when it has greater lead time, the greater propensity for them to cancel their current order.
3g) On Deposit Type¶
In [131]:
sns.set(style = "whitegrid")
plt.figure(figsize= (18,5))
# countplot for deposit type count
plt.subplot(1,3,1)
plt.title("Deposit Type Count", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="deposit_type")
# countplot for cancellation count wth cancellation hue
plt.subplot(1,3,2)
plt.title("Cancellation Count by Deposit Type", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="deposit_type", hue='is_canceled')
# barplot for deposit type & cancellation
plt.subplot(1,3,3)
plt.title("Cancellation Mean by Deposit Type", fontdict={'fontsize': 15})
sns.barplot(data=mydata, y="is_canceled", x='deposit_type')
Out[131]:
In [136]:
# crosstab for deposit type & order cancellation
pd.crosstab(mydata['deposit_type'],mydata['is_canceled'])
Out[136]:
In [104]:
# lead_time distribution by deposit_type
sns.set_style("whitegrid")
sns.FacetGrid(data=mydata, col='deposit_type').map(sns.distplot,'lead_time').add_legend()
Out[104]:
In [97]:
# looking at non refund deposits yet cancelling orders
NonRefund_1 = mydata[(mydata['deposit_type']=='Non Refund') & (mydata['is_canceled']==1)]
NonRefund_1.describe().T
Out[97]:
From the observations:
- In terms of deposit type count, it seems that most of the customers (87,6%) prefer bookings with No Deposit while only little of them prefers Refundable type (0,1%). No wonder that most order cancellation counts is within the No Deposit type.
- Counterintuitively, the plot shows that the Non Refund type has the highest average of order cancellations. Interestingly, it does have a significantly smaller stayed order (93 orders) compared ot its canceled orders (14,480 orders). In other words, most of the customers who chooses Non Refund canceled their orders. Very counterintuitive.
- It appears that most customers who canceled their Non Refund deposits has lead_time of 103++ days (gap until arrival time) with median of 186 days & average of 213 days. These are big lead_time numbers. As we can see from our previous analysis, orders with bigger lead_time (at least after 50++ days) has a bigger propensity to cancel. Perhaps, this is partly why most of the Non Refund deposits are canceled.
3h) On Arrival Weeks & Months¶
In [9]:
# countplot by week
sns.set(style = "whitegrid")
plt.figure(figsize=(20,6))
sns.countplot(mydata['arrival_date_week_number'])
plt.title("Room orders over the year (in weeks)", fontsize=16)
plt.xlabel("Week", fontsize=16)
plt.xticks(rotation=45)
plt.ylabel("Orders", fontsize=16)
plt.show()
# ordering data by month:
monthly_adr = mydata[["hotel", "arrival_date_month", "adr","is_canceled"]].sort_values("arrival_date_month")
ordered_months = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"]
monthly_adr["arrival_date_month"] = pd.Categorical(monthly_adr["arrival_date_month"], categories=ordered_months, ordered=True)
# barplot with standard deviation:
plt.figure(figsize=(20, 6))
sns.lineplot(x = "arrival_date_month", y="adr", hue="hotel", data=monthly_adr,
hue_order = ["City Hotel", "Resort Hotel"], ci="sd", size="hotel", sizes=(2.5, 2.5))
plt.title("Room prices over the year (in months)", fontsize=16)
plt.xlabel("Month", fontsize=16)
plt.xticks(rotation=45)
plt.ylabel("Average Daily Rate [EUR]", fontsize=16)
plt.show()
# countplot by month
plt.figure(figsize=(20,6))
# sns.countplot(data=mydata, x="deposit_type", hue='is_canceled')
sns.countplot(data=monthly_adr, x="arrival_date_month" , hue="is_canceled")
plt.title("Order Cancellations over the year (in months)", fontsize=16)
plt.xlabel("Month", fontsize=16)
plt.xticks(rotation=45)
plt.ylabel("Orders", fontsize=16)
plt.show()
From the observations:
- In terms of room demand, we can see that in the 30th week until 34th week there's a surge in room bookings, with 33rd week having the highest order count. We can also see that the surge in demand is also translated to higher room prices or average daily rates (adr), especially in the Resort Hotel type. As we all know, this is to no surprise because this surge is happening within the Summer season.
- In terms of order cancellations, we can see that the canceled order amount increases/decreases as there are increases/decreases in room orders (in this case, stayed orders). Nevertheless, we could stil see that the highest surge happens within the Summer months.
3i) On Customer Type¶
In [5]:
sns.set(style = "whitegrid")
plt.figure(figsize= (18,5))
# countplot for deposit type count
plt.subplot(1,3,1)
plt.title("Customer Type Count", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="customer_type")
# countplot for cancellation count wth cancellation hue
plt.subplot(1,3,2)
plt.title("Cancellation Count by Customer Type", fontdict={'fontsize': 15})
sns.countplot(data=mydata, x="customer_type", hue='is_canceled')
# barplot for deposit type & cancellation
plt.subplot(1,3,3)
plt.title("Cancellation Mean by Customer Type", fontdict={'fontsize': 15})
sns.barplot(data=mydata, y="is_canceled", x='customer_type')
Out[5]:
In [6]:
# crosstab for deposit type & order cancellation
pd.crosstab(mydata['customer_type'],mydata['is_canceled'])
Out[6]:
In [105]:
# lead_time distribution by customer_type
sns.set_style("whitegrid")
sns.FacetGrid(data=mydata, col='customer_type').map(sns.distplot,'lead_time').add_legend()
Out[105]:
In [100]:
# looking at contract customers yet cancelling orders
Contract_1 = mydata[(mydata['customer_type']=='Contract') & (mydata['is_canceled']==1)]
Contract_1.describe().T
Out[100]:
From the observations:
- In terms of customer type, it seems that the largest part of them are transient types, both individuals & groups. It is to no surprise that that these transient types contribute the most on order cancellation counts. What's surprising is that there are still orders, although a few, that is made by contract types yet canceled.
- After we see the lead_time mean & median of the Contract group, it shows 221 days & 278 days respectively. Again, these are big lead_time numbers, and perhaps it might partly contribute to this group's bigger propensity to cancel
3j) On Distributions¶
In [106]:
from scipy.stats import shapiro, anderson, skew, kurtosis
for i in mydata.select_dtypes(exclude = 'object').columns:
print("[{}]".format(i))
if shapiro(mydata[i])[1] < 0.05:
print('Not-Normal Distribution')
else:
print('Normal Distribution')
print('Skewness: {}, Kurtosis {}'.format(skew(mydata[i]), kurtosis(mydata[i])))
sns.distplot(mydata[i])
plt.title(i)
plt.show()
print('\n')
From the observations:
- In terms of data distribution, we could see from our Shapiro Test that all our features has values below 0,05. In other words, none of the numerical features has a normal/Gaussian distribution
4. MODEL DEVELOPMENT¶
4a) Data Preprocessing¶
In [54]:
# making country names into domestic & international
def tourist(cols):
if (cols == 'PRT'):
cols = 'Domestic'
else:
cols = 'International'
return cols
mydata['country'] = mydata['country'].apply(tourist)
mydata['country'].value_counts().head(10)
Out[54]:
In [55]:
# streamlining the deposit type into two categories only (0=Flexible; 1=Inflexible)
def yes(cols):
if (cols == "No Deposit") or (cols == "Refundable"):
cols = 0
else:
cols = 1
return cols
mydata['deposit_type'] = mydata['deposit_type'].apply(yes)
mydata['deposit_type'].value_counts()
Out[55]:
4b) Model Preparation¶
In [58]:
# taking relevant features into model
mycoba = mydata[['hotel','adr','country','lead_time','previous_cancellations','previous_bookings_not_canceled','required_car_parking_spaces','total_of_special_requests',
'children','adults','customer_type','deposit_type','market_segment','is_repeated_guest','is_canceled']]
In [59]:
# model features
mycoba.info()
In [85]:
# making dummies for selected categorical columns
mydummy = pd.get_dummies(data= mycoba, drop_first= True, columns = ['hotel','country','customer_type','deposit_type','market_segment'] )
mydummy.columns
Out[85]:
In [61]:
# assigning features
IV = ['adr', 'lead_time', 'previous_cancellations',
'previous_bookings_not_canceled', 'required_car_parking_spaces',
'total_of_special_requests', 'children', 'adults', 'is_repeated_guest',
'hotel_Resort Hotel', 'country_International',
'customer_type_Group', 'customer_type_Transient',
'customer_type_Transient-Party', 'deposit_type_1',
'market_segment_Complementary', 'market_segment_Corporate',
'market_segment_Direct', 'market_segment_Groups',
'market_segment_Offline TA/TO', 'market_segment_Online TA']
# independent variables
x = mydummy[IV]
# dependent/target variable
y = mydummy['is_canceled']
In [10]:
# splitting the training data - test data = 80% : 20%
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state= 101)
In [11]:
# create a new x_train & y_train variable (x_trainres & y_trainres) that is resampled using SMOTE method
from imblearn.over_sampling import SMOTE
from collections import Counter
y_train = y_train.astype('int')
smo = SMOTE(random_state=0, sampling_strategy='minority')
x_trainres, y_trainres = smo.fit_resample(x_train, y_train)
print("normal data (target) :",sorted(Counter(y_train).items()))
print("oversampled data (target) :",sorted(Counter(y_trainres).items()))
4c) Model Comparison¶
In [14]:
# importing libraries & algorithms to be compared
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, fbeta_score
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import BernoulliNB
from sklearn.svm import SVC
from xgboost import XGBClassifier
# to feed the random state
seed = 7
------------- ON ACCURACY -------------¶
In [20]:
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('DT', DecisionTreeClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('NB', BernoulliNB()))
models.append(('XGB', XGBClassifier()))
models.append(('LR2', LogisticRegression()))
models.append(('DT2', DecisionTreeClassifier()))
models.append(('RF2', RandomForestClassifier()))
models.append(('NB2', BernoulliNB()))
models.append(('XGB2', XGBClassifier()))
# scoring
results = []
names = []
scoring = 'accuracy'
# evaluate each model in turn for normal & oversampled data
for name, model in models:
kfold = KFold(n_splits=10, random_state=seed)
if "2" in name:
cv_results = cross_val_score(model, x_trainres, y_trainres, cv=kfold, scoring=scoring)
else:
cv_results = cross_val_score(model, x_train, y_train, cv=kfold, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
# boxplot algorithm comparison for normal & oversampled data
fig = plt.figure(figsize=(20,6))
fig.suptitle('Algorithm Comparison - On Accuracy')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
- The highest overall Accuracy score is achieved by RF2 (RandomForestClassifier-Oversampled) with an average score of 0.876
- The second highest overall Accuracy score is achieved by XGB2 (XGBoostClassifier-Oversampled) with an average score of 0.858
------------- ON RECALL -------------¶
In [25]:
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('DT', DecisionTreeClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('NB', BernoulliNB()))
models.append(('XGB', XGBClassifier()))
models.append(('LR2', LogisticRegression()))
models.append(('DT2', DecisionTreeClassifier()))
models.append(('RF2', RandomForestClassifier()))
models.append(('NB2', BernoulliNB()))
models.append(('XGB2', XGBClassifier()))
# scoring
results = []
names = []
scoring = 'recall'
# evaluate each model in turn for normal & oversampled data
for name, model in models:
kfold = KFold(n_splits=10, random_state=seed)
if "2" in name:
cv_results = cross_val_score(model, x_trainres, y_trainres, cv=kfold, scoring=scoring)
else:
cv_results = cross_val_score(model, x_train, y_train, cv=kfold, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
# boxplot algorithm comparison for normal & oversampled data
fig = plt.figure(figsize=(20,6))
fig.suptitle('Algorithm Comparison - On Recall')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
- The highest overall Recall score is achieved by RF2 (RandomForestClassifier-Oversampled) with an average score of 0.852
- The second highest overall Recall score is achieved by DT2 (DecisionTreeClassifier-Oversampled) with an average score of 0.836
------------- ON F1 SCORE -------------¶
In [14]:
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('DT', DecisionTreeClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('NB', BernoulliNB()))
models.append(('XGB', XGBClassifier()))
models.append(('LR2', LogisticRegression()))
models.append(('DT2', DecisionTreeClassifier()))
models.append(('RF2', RandomForestClassifier()))
models.append(('NB2', BernoulliNB()))
models.append(('XGB2', XGBClassifier()))
# scoring
results = []
names = []
scoring = 'f1'
# evaluate each model in turn for normal & oversampled data
for name, model in models:
kfold = KFold(n_splits=10, random_state=seed)
if "2" in name:
cv_results = cross_val_score(model, x_trainres, y_trainres, cv=kfold, scoring=scoring)
else:
cv_results = cross_val_score(model, x_train, y_train, cv=kfold, scoring=scoring)
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
# boxplot algorithm comparison for normal & oversampled data
fig = plt.figure(figsize=(20,6))
fig.suptitle('Algorithm Comparison - On F1 Score')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
- The highest overall F1 score is achieved by RF2 (RandomForestClassifier-Oversampled) with an average score of 0.846
- The second highest overall F1 score is achieved by XGB2 (XGBoostClassifier-Oversampled) with an average score of 0.828
------------- ON ROC-AUC -------------¶
In [13]:
# prepare models
models = []
models.append(('LR', LogisticRegression()))
models.append(('DT', DecisionTreeClassifier()))
models.append(('RF', RandomForestClassifier()))
models.append(('NB', BernoulliNB()))
models.append(('XGB', XGBClassifier()))
models.append(('LR2', LogisticRegression()))
models.append(('DT2', DecisionTreeClassifier()))
models.append(('RF2', RandomForestClassifier()))
models.append(('NB2', BernoulliNB()))
models.append(('XGB2', XGBClassifier()))
# scoring
results = []
names = []
# evaluate each model in turn for normal & oversampled data
for name, model in models:
if "2" in name:
cv_results = cross_val_score(model, x_trainres, y_trainres, cv=10, n_jobs=-3, scoring = 'roc_auc')
else:
cv_results = cross_val_score(model, x_train, y_train, cv=10, n_jobs=-3, scoring= 'roc_auc')
results.append(cv_results)
names.append(name)
msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std())
print(msg)
# boxplot algorithm comparison for normal & oversampled data
fig = plt.figure(figsize=(20,6))
fig.suptitle('Algorithm Comparison - On ROC-AUC')
ax = fig.add_subplot(111)
plt.boxplot(results)
ax.set_xticklabels(names)
plt.show()
- The highest overall ROC-AUC score is achieved by RF2 (RandomForestClassifier-Oversampled) with an average score of 0.951
- The second highest overall ROC-AUC score is achieved by XGB2 (XGBoostClassifier-Oversampled) with an average score of 0.944
4d) Model Selection & Description¶
- In overall, the model that shows the top most consistent performances across parameters, especially in Recall score, is the RF2 (RandomForestClassifier-Oversampled) model. Therefore, we will use this model for further analysis & hyperparameter tuning.
-------------------------- RANDOM FOREST CLASSIFIER ----------------------------¶
In [30]:
# model fitting for normal data
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(random_state=7)
rfc.fit(x_train, y_train)
Out[30]:
In [31]:
# model fitting for oversampled data
from sklearn.ensemble import RandomForestClassifier
rfc2 = RandomForestClassifier(random_state=7)
rfc2.fit(x_trainres, y_trainres)
Out[31]:
In [41]:
# importing relevant libraries
from sklearn.metrics import classification_report, confusion_matrix, roc_curve, roc_auc_score
# normal - classification report for train & test data
print("### TRAIN DATA (NORMAL)")
print(classification_report(y_train, rfc.predict(x_train)))
rfc_pred = rfc.predict(x_test)
print("### TEST DATA (NORMAL)")
print(classification_report(y_test, rfc_pred))
# oversampled - classification report for train & test data
print("### TRAIN DATA (OVERSAMPLED)")
print(classification_report(y_trainres, rfc2.predict(x_trainres)))
rfc_pred2 = rfc2.predict(x_test)
print("### TEST DATA (OVERSAMPLED)")
print(classification_report(y_test, rfc_pred2))
In [42]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, rfc_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Normal')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
# confusion matrix for oversampled data
cnf_matrix = confusion_matrix(y_test, rfc_pred2)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - Oversampled')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [32]:
# ROC-AUC Score
from sklearn import metrics
for i in range(2):
if i == 0:
pred_proba = rfc.predict_proba(x_test)[::,1]
elif i == 1:
pred_proba = rfc2.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, pred_proba)
auc = metrics.roc_auc_score(y_test, pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
if i == 0:
plt.title('Receiver Operator Characteristic - Normal')
elif i == 1:
plt.title('Receiver Operator Characteristic - Oversampled')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
From the observations:
- We could see that the outcomes of RF2 (RandomForestClassifier-Oversampled) & RF (RandomForestClassifier-Normal) are both equally good. Their Accuracy & ROC-AUC scores are very high, this is a good thing. Their Accuracy scores of >= 0.85 means that 85% of the total events are predicted with the right/actual labels. And their ROC-AUC scores of 0.93 means that the models have a 93% probability of distinguishing between cancelling & staying orders.
- Nevertheless, since in this case we're trying to minimize incidents of False Negatives (labeled canceling orders as staying orders), we should give prioritization & optimization on Recall score. In terms of Recall of cancelling orders, RF2 has the higher score of and should be the one used for predicting cancelling orders.
In [89]:
# feature importance for normal data
rfc_coef1 = pd.Series(rfc.feature_importances_, x_train.columns).sort_values(ascending= False)
rfc_coef1.plot(kind = 'bar', title='Feature Importances - Normal Data')
plt.show()
# feature importance for oversampled data
rfc_coef2 = pd.Series(rfc2.feature_importances_, x_trainres.columns).sort_values(ascending= False)
rfc_coef2.plot(kind = 'bar', title='Feature Importances - Oversampled Data')
plt.show()
From the observations:
- From feature importance we can see which features that contribute to the most information gained in predicting our target. Those top features that are consistently shown in the normal & oversampled data is lead_time (amount of days between booking & arrival date) & adr (average daily rate). Special notice to country, deposit_type & total_of_special_requests feature that appears to approximate the top features in both data. In other words, all of these features are important for our model in predicting cancelling orders.
In [109]:
# permutation importance for normal data
from eli5 import show_weights
from eli5.sklearn import PermutationImportance
rfcperm = PermutationImportance(rfc, scoring = 'roc_auc', random_state= 7).fit(x_test, y_test)
show_weights(rfcperm, feature_names = list(x_test.columns))
Out[109]:
In [110]:
# permutation importance for oversampled data
rfcperm2 = PermutationImportance(rfc2, scoring = 'roc_auc', random_state= 7).fit(x_test, y_test)
show_weights(rfcperm2, feature_names = list(x_test.columns))
Out[110]:
From the observations:
- From the permutation feature importance, we can see the upper features (green) that contribute the most to the drop of the overall model scores if the feature's values are randomly shuffled. Those top features that are consistent on the top (in the darkest of green shades) in both normal & oversampled data are country, lead_time, adr, total_of_special_requests & deposit_type. Special notice for market_segment that came amongst the top in both data. In other words, all of these features are important for our model in predicting cancelling orders.
-------------------------- RANDOM FOREST CLASSIFIER (HYPERPARAMETER TUNING) ----------------------------¶
In [43]:
## GRIDSEARCHCV
from sklearn.model_selection import GridSearchCV
grid1 = GridSearchCV(estimator = rfc2,
refit = 'recall',
param_grid = {'random_state': np.arange(1,101)},
scoring = 'recall',
cv = 5, n_jobs = -1)
In [44]:
grid1.fit(x_trainres, y_trainres)
Out[44]:
In [46]:
# best parameters
grid1.best_params_
Out[46]:
In [41]:
# model fitting for oversampled data
from sklearn.ensemble import RandomForestClassifier
rfc3 = RandomForestClassifier(random_state=16)
rfc3.fit(x_trainres, y_trainres)
Out[41]:
In [50]:
# oversampled - classification report for train & test data
print("### TRAIN DATA (OVERSAMPLED)")
print(classification_report(y_trainres, rfc3.predict(x_trainres)))
rfc_pred3 = rfc3.predict(x_test)
print("### TEST DATA (OVERSAMPLED)")
print(classification_report(y_test, rfc_pred3))
In [51]:
# confusion matrix
cnf_matrix = confusion_matrix(y_test, rfc_pred3)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - RF3')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [52]:
# ROC-AUC Score
from sklearn import metrics
pred_proba = rfc3.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, pred_proba)
auc = metrics.roc_auc_score(y_test, pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - RF3')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
In [49]:
grid2 = GridSearchCV(estimator = rfc2,
refit = 'recall',
param_grid = {
'n_estimators':[1000,1800],
'bootstrap': [True,False],
'max_features': ['sqrt','auto'],
'max_depth': [80,None],
},
scoring = 'recall',
cv = 5, n_jobs = -1)
In [50]:
grid2.fit(x_trainres, y_trainres)
Out[50]:
In [53]:
# best parameters
grid2.best_params_
Out[53]:
In [45]:
# model fitting for oversampled data
from sklearn.ensemble import RandomForestClassifier
rfc4 = RandomForestClassifier(bootstrap=True, max_depth=80, max_features= 'sqrt', n_estimators= 1000)
rfc4.fit(x_trainres, y_trainres)
Out[45]:
In [47]:
# oversampled - classification report for train & test data
print("### TRAIN DATA (OVERSAMPLED)")
print(classification_report(y_trainres, rfc4.predict(x_trainres)))
rfc_pred4 = rfc4.predict(x_test)
print("### TEST DATA (OVERSAMPLED)")
print(classification_report(y_test, rfc_pred4))
In [48]:
# confusion matrix
cnf_matrix = confusion_matrix(y_test, rfc_pred4)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - RF4')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [49]:
# ROC-AUC Score
from sklearn import metrics
pred_proba = rfc4.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, pred_proba)
auc = metrics.roc_auc_score(y_test, pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - RF4')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
From the observations:
- We could see that our trials of hyperparameter tuning haven't generated much improvement. Nevertheless, special notes on both tuned models that they generate less & less False Negatives than our non-tuned RF2, from 1689 cases (RF2) lowered to 1673 cases (RF3) and finally to 1664 cases (RF4) of False Negatives (with levels of Accuracy & ROC-AUC score staying the same).
- We should then try to build our autoML model for our the next step of comparison.
-------------------------- AUTOML CLASSIFIER ----------------------------¶
In [12]:
# let's benchmark our tuned model performance with AutoML
from tpot import TPOTClassifier
tpot = TPOTClassifier(subsample = 0.8, verbosity = 2, warm_start=True, early_stop=20, max_time_mins= 60, n_jobs= -2)
In [13]:
# fitting TPOT to our data
tpot.fit(x_train, y_train)
Out[13]:
In [14]:
# exporting our model results
tpot.export('tpot_HOTEL.py')
In [22]:
# using our exported AutoML model ('tpot_HOTEL.py') to our data
import numpy as np
import pandas as pd
from sklearn.decomposition import FastICA
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
# fitting the AutoML for our data
autoML = make_pipeline(
FastICA(tol=0.0),
KNeighborsClassifier(n_neighbors=57, p=1, weights="distance"))
autoML.fit(x_train, y_train)
Out[22]:
In [34]:
# normal - classification report for train & test data
print("### TRAIN DATA")
print(classification_report(y_train, autoML.predict(x_train)))
autoML_pred = autoML.predict(x_test)
print("### TEST DATA")
print(classification_report(y_test, autoML_pred))
In [36]:
# confusion matrix for normal data
cnf_matrix = confusion_matrix(y_test, autoML_pred)
sns.heatmap(pd.DataFrame(cnf_matrix), annot=True, fmt='g')
plt.tight_layout()
plt.title('Confusion Matrix - AutoML')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
plt.show()
In [37]:
# ROC-AUC Score
from sklearn import metrics
pred_proba = autoML.predict_proba(x_test)[::,1]
fpr, tpr, _ = metrics.roc_curve(y_test, pred_proba)
auc = metrics.roc_auc_score(y_test, pred_proba)
roc_auc = metrics.auc(fpr, tpr)
plt.figure(figsize= (10,5))
plt.title('Receiver Operator Characteristic - AutoML')
plt.plot(fpr,tpr,label="DT, AUC="+str(auc))
plt.plot(fpr, tpr, 'b', label= 'Rounded AUC = {}'.format(round(roc_auc,2)))
plt.legend(loc = 'lower right')
plt.plot([0,1], [0,1], 'r--')
plt.xlim([0,1])
plt.ylim([0,1])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.legend(loc=4)
plt.show()
From the observations:
- We could see that our trials of autoML haven't generated much improvement as well. Although it has slightly higher levels of F1 Score, it shows lower levels of Recall score on our cancelling orders. Its level of False Negatives is even higher than our non-tuned RF2, from 1689 cases (RF2) increased to 1915 cases (autoML). It seems that our RF2 & tuned RF2 (RF3 & RF4) performs better than the autoML model
4e) Learning Curve¶
- From the previous analysis, we have seen that RF2 (RandomForestClassifier-Oversampled) has good performance but RF4 (RandomForestClassifier-OversampledTuned) is slightly better. These models are also better than the autoML model generated. Therefore, we will use both of this model to check their learning curves.
In [40]:
# RF2
from sklearn.model_selection import learning_curve
train_sizes, train_scores, test_scores = learning_curve(estimator=rfc2,
X=x,
y=y,
train_sizes=np.linspace(0.5, 1.0, 5),
cv=10)
# Mean value of accuracy against training data
train_mean = np.mean(train_scores, axis=1)
print('train mean: ')
print(train_mean)
# Standard deviation of training accuracy per number of training samples
train_std = np.std(train_scores, axis=1)
# Same as above for test data
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)
print('test mean: ')
print(test_mean)
# Plot training accuracies
plt.plot(train_sizes, train_mean, color='red', marker='o', label='Training Accuracy')
# Plot the variance of training accuracies
plt.fill_between(train_sizes,
train_mean + train_std,
train_mean - train_std,
alpha=0.15, color='red')
# Plot for test data as training data
plt.plot(train_sizes, test_mean, color='blue', linestyle='--', marker='s',
label='Test Accuracy')
plt.fill_between(train_sizes,
test_mean + test_std,
test_mean - test_std,
alpha=0.15, color='blue')
plt.xlabel('Number of training samples')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
In [53]:
# RF4
train_sizes, train_scores, test_scores = learning_curve(estimator=rfc4,
X=x,
y=y,
train_sizes=np.linspace(0.5, 1.0, 5),
cv=10)
# Mean value of accuracy against training data
train_mean = np.mean(train_scores, axis=1)
print('train mean: ')
print(train_mean)
# Standard deviation of training accuracy per number of training samples
train_std = np.std(train_scores, axis=1)
# Same as above for test data
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)
print('test mean: ')
print(test_mean)
# Plot training accuracies
plt.plot(train_sizes, train_mean, color='red', marker='o', label='Training Accuracy')
# Plot the variance of training accuracies
plt.fill_between(train_sizes,
train_mean + train_std,
train_mean - train_std,
alpha=0.15, color='red')
# Plot for test data as training data
plt.plot(train_sizes, test_mean, color='blue', linestyle='--', marker='s',
label='Test Accuracy')
plt.fill_between(train_sizes,
test_mean + test_std,
test_mean - test_std,
alpha=0.15, color='blue')
plt.xlabel('Number of training samples')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
From the observations:
- We could see that the test accuracy scores of both RF models (RF2 &RF4) are lower than the train scores. The test accuracy scores have reached levels of 0.7 having the train scores of near 1, this is good actually. However, seeing from the generalization gaps, we can see that our models has a predisposition to overfit tha data. Nevertheless, our models still performs excellently in the previous test:train data of 20%:80%. Perhaps, more samples could be given to the model to see what it could achive further. All in all, both model is still usable with RF4 being more preferable.
In [103]:
# saving algorithm for further usage
import pickle
filename = 'hotel_cancellation_tuned.sav';
pickle.dump(rfc4, open(filename, 'wb'))
5. CONCLUDING NOTES¶
1) On features:¶
Through all the observations, there are at many features that keeps showing significance and they are lead_time, adr, market_Ssegment, country, previous_cancellations and deposit_type features, with lead_time perhaps as the most important feature in relation to cancelling orders ('is_canceled'). In other words, these features could be the features that has better explanatory relations towards our target variable of cancelling orders and may prove to be important for further observations & practical applications.
2) On predictions:¶
In overall, the best model for our prediction is RandomForestClassifier-Oversampled that's hyperparameter-tuned (RF4). It even gives better results than the autoML model, especially in its Recall score since we're trying to minimize False Negatives in our predictions. This model may be useful for further descriptions & deployments.